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pproximately 30 years have passed since Herman Goldstein (1979) first articulated the 
basic principles of problem-oriented policing. During those years, problem-oriented 
olicing emerged as a widely practiced approach to crime prevention in police depart- 
ments throughout the United States as well as throughout the world. Problem-oriented policing 
enjoys broad support from federal agencies, professional policing groups, and a small cadre of 
scholars interested in effective crime prevention practices. Given this support, it was surprising 
to learn that Weisburd, Telep, Hinkle and Eck (2010, this issue) identified only ten evaluations 
that met the minimum methodological standards of their systematic review—a comparison 
group that did not receive the problem-oriented policing treatment. Their systematic review also 
identified 45 before—after intervention studies without a strong control or comparison group. 
To some observers, a ratio of one strong evaluation to every four or five weak evaluations 
might not sound overly concerning. Weak evaluations, unfortunately, provide less valid answers 
to policy questions when compared with well-designed quasi-experiments and randomized 
controlled trials (Campbell and Stanley, 1966; Cook and Campbell, 1979). Several crime and 
justice scholars suggest that a “moral imperative” exists in pursuing the most rigorous evalua- 
tion designs to discover whether a program is effective (e.g., Boruch, 1975; Weisburd, 2003). 
Isolating the effects of treatments or programs from other confounding aspects of selection 
or design is viewed as one of the evaluator’s most important obligations to society. When the 
evaluation evidence base is informed largely by weak designs, practitioners risk implementing 
certain treatments or programs as effective crime prevention practices when they are not, which 
can lead to significant economic and social costs. 
Much of the academic work on problem-oriented policing seeks to improve practice by refin- 


ing key steps in the process such as encouraging the in-depth analysis of problems and searching 
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for innovative responses that go beyond traditional enforcement. Unfortunately, although these 
academics press practitioners to conduct more rigorous assessments of implemented prevention 
strategies, they also tend to be dismissive of the use of more rigorous research designs, such as 
randomized controlled trials, in evaluating the crime prevention effects of problem-oriented 
policing initiatives. This apparent bias against more rigorous designs is certainly a contributing 
factor to the weak scientific evidence base for the problem-oriented policing approach. Beyond 
increasing the openness of problem-oriented policing scholars to include stronger research designs, 
investments need to be made to develop strong working relationships with police practitioners 


so that opportunities can be created to conduct more rigorous evaluations. 


Can Problem-Oriented Policing Be Properly 

Evaluated Using More Rigorous Designs? 

The problem-oriented policing movement has benefited greatly from the hard work of a few 
academics, such as Herman Goldstein, Ronald V. Clarke, Nick Tilley, John E. Eck, Johannes 
Knutsson, and others who are concerned about the quality and effectiveness of policing. These 
academics have made a concerted effort to improve the quality of problem analysis, identify 
alternative approaches to deal with the underlying conditions and situations that cause recur- 
ring problems to exist, as well as disseminate examples of successful problem-oriented policing 
projects through conference participation, the production of practitioner-friendly guides, and 
strategic partnerships with governmental agencies such as the U.S. Office of Community- 
Oriented Policing Services. Indeed, these scholars deserve much credit for stimulating interest 
in the problem-oriented policing approach in police departments around the world. 

These academics are vocal in lamenting the poor quality of practitioner-led assessments 
of problem-oriented policing projects and conclude that police departments need consider- 
able support from academic partners to carry out evaluations (e.g., Clarke, 1998). Although 
they advocate for a range of evaluation methodologies, these academics are often skeptical of 
using more rigorous evaluation designs to evaluate problem-oriented policing projects (Eck, 
2002; Knutsson, 2009; Pawson and Tilley, 1997; Tilley, 2009). Randomized controlled trials, 
in particular, are described in a discouraging manner. They suggest that problem-oriented 
policing interventions are highly customized for specific local problems and, as such, often are 
not amenable to rigorous designs that rule out threats to internal validity. Experimental evalu- 
ations of problem-oriented policing also are described as “academic research projects” with low 
external validity that does not reflect the reality of problem-oriented policing as it is practiced 
in the field (Knutsson, 2009: 20). 

These problem-oriented policing scholars suggest that single group before—after designs 
are the modal assessment methodology to evaluate problem-oriented policing interventions, 
and they argue that it is much more feasible and fruitful to decrease threats to internal validity 
through different means such as adding qualitative evidence, using time-series data, investigating 
possible alternative casual hypotheses, or being guided by precise theory (Eck, 2002; Knutsson, 
2009; Pawson and Tilley, 1997; Tilley, 2009). This pragmatic view has generated compelling 
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case studies and a growing body of practical evidence that supports problem-oriented policing 
as a promising approach to crime prevention (Skogan and Frydl, 2004; Weisburd and Eck, 
2004). Unfortunately, this practical approach to generate knowledge on improved police crime 
prevention practices is problematic in regard to generating more rigorous scientific evidence 
in the long term. If academics interested in exploring the effects of problem-oriented policing 
on crime are not being encouraged by their peers to use more rigorous research designs, then 
problem-oriented policing will never have a strong evaluation research base. As “evidence- 
based” approaches, with their premium on rigorous evaluation designs, gain currency in policy 
circles, funding for continued expansion of the problem-oriented policing agenda might be 
threatened. 

In reality, fewer police departments will be willing to host more intrusive research designs 
such as randomized controlled trials. As Eck (2002) described, there are many reasons why police 
departments might not be willing to play host to an experiment—the department might have 
other pressing business, the department's leadership might not be tested, and other reasons. 
When a police department is willing to engage a more rigorous design, the external validity 
of the findings are called into question because the host departments and their contexts are 
described as not representative of all agencies or contexts that use the intervention. Drawing 
on Campbell and Stanley (1966), Eck (2002: 104) identified this issue as a problem of the 
“interaction of selection and treatment.” 

Although this point is legitimate, all problem-oriented policing evaluations suffer from 
external validity concerns regardless of the degree of internal rigor in the evaluation research 
design. Problem-oriented policing is primarily an analytic approach to crime prevention that 
requires customizing interventions to highly localized crime and disorder problems. What works 
in preventing a street robbery problem in the public areas of Harvard Square in Cambridge, 
Massachusetts, might not work when applied to repeated robberies occurring in the London 
Underground subway system. Appropriate interventions need to be applied in both contexts that 
are based on careful analysis of the conditions that create the compelling criminal opportunities. 
Neither the evaluation findings of a carefully constructed single group before—after design nor 
the findings of a randomized controlled experiment will travel perfectly across these settings. 
Problem-oriented policing evaluations of many forms provide valuable guidance to police officers 
struggling with real-world problems. However, given the highly customized nature of effective 
problem-oriented policing interventions, it is important to recognize that the generalizability of 
specific crime prevention practices identified in an effective application of the approach might 
be limited, regardless of the evaluation approach used. 

Randomized controlled trials and more rigorous quasi-experimental designs can be developed 
to maximize their external validity. A well-designed test can collect the necessary qualitative 
and quantitative data to understand the processes that underlie problem-oriented policing and 
“why” it might impact crime and disorder problems. For instance, the units of analysis in the 
evaluation design—whether they are problem places, problem people, hot products, or some 


other unit—must be analyzed thoroughly to ensure equivalent treatment and control groups. 
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Although this task is time consuming, it is by no means insurmountable through established 
matching techniques (that range from descriptive exercises to more formal propensity score 
analyses) and subsequent allocation from like pairs or statistically comparable blocks to treat- 
ment and control conditions. The thorough description of problem units provides police 
practitioners with important information regarding the problems addressed in the study so 
that they can determine whether any effective interventions are applicable to problems faced 
in their local jurisdictions. 

Rigorous evaluation designs can accommodate the reality of problem-oriented policing as 
practiced in the field. An important consideration in gaining real-world insights on the crime 
prevention value of problem-oriented policing in dealing with crime and disorder problems is 
to test the general approach rather than any specific set of tactics. In two randomized controlled 
experiments (Braga and Bond, 2008; Braga, Weisburd, Waring, Mazerolle, Spelman, and 
Gajewski, 1999), the treatment was the meta-method known as “problem-oriented policing,” 
which comprised several specific operational tactics implemented by the officers to control the 
identified problems at treatment places.' Careful quantitative and qualitative documentation of 
the content of these problem-oriented interventions provided police practitioners with a clear 
sense of the types of responses to address problem places so that they can develop knowledge 
on a range of strategies that could be plausibly effective in their jurisdictions. Indeed, both 
experiments documented interventions that closely resembled problem-oriented policing as 
it is usually practiced in the field—“shallow” problem solving characterized by weak problem 
analysis and limited response development. 

When interventions are described in sufficient detail, further analyses, such as mediation 
analysis, can be used to determine the key elements of the intervention that seem to be driv- 
ing any observed crime control gains. Using this approach, a randomized controlled trial of 
problem-oriented policing in Lowell, Massachusetts, found that the strongest crime prevention 
gains at crime and disorder hot spots were driven by situational prevention strategies rather 
than by misdemeanor arrest strategies or short-term, police-led social service strategies (Braga 
and Bond, 2008). Because many jurisdictions deal with crime and disorder hot spot locations, 
the basic findings of this study provide important guidance to police executives interested in 


maximizing their ability to manage similar problem places. 


Creating the Opportunity to Conduct More Rigorous Evaluations 

Academics must work to create the opportunities to test problem-oriented policing interven- 
tions using more rigorous designs with higher degrees of internal validity. Unfortunately, past 
partnerships between academics and police practitioners often are characterized by role conflicts 
such as researchers reporting the “bad news” that an evaluated program was not effective in 


preventing crime (Weisburd, 1994). For academic researchers, success or failure matters less 


1. The Braga and Bond (2008) randomized controlled trial was completed after Weisburd et al. (2010) finished 
their systematic search for eligible problem-oriented policing studies and, as such, is not included in their 
review. 
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than their commitment to the development of knowledge of what does or does not work in 
preventing crime. For police practitioners, this news could be interpreted as their personal 
failure, and the skepticism of academics could be viewed as irritating. The police also can view 
traditional research and evaluation roles played by academics (often involving data collection 
and analysis afier programs have been developed) as not particularly helpful in their efforts to 
prevent crime. 

In recent years, however, the demand for the participation of academic researchers in 
problem-oriented policing projects has increased because the police have come to recognize the 
importance of strategic information products in developing effective crime prevention interven- 
tions. Academics can help the police by conducting research on urban crime problems to focus 
limited enforcement, intervention, and prevention resources on high-risk offenders, victims, and 
places. Problem-oriented interventions based on research insights have been associated with a 
60% reduction in youth homicide in Boston (Braga, Kennedy, Waring, and Piehl, 2001) and 
a 40% total homicide reduction in Indianapolis (McGarrell, Chermak, Wilson, and Corsaro, 
2006). These success stories have made academic researchers an important part of new crime 
prevention initiatives. For example, the U.S. Department of Justice-sponsored Project Safe 
Neighborhoods (PSN) initiative provided each of the 94 U.S. Attorney’s districts with funds to 
hire academic research partners to help understand and address serious gun violence problems 
in local jurisdictions. A recent national evaluation of PSN found that treatment cities with high 
levels of implementation, which include in-depth problem analysis research to tailor prevention 
strategies, were associated with declines in violent crime (McGarrell et al., 2009). 

The increasing police openness to the assistance of academics in understanding the nature 
of crime problems provides an important opportunity to develop higher quality evaluations of 
problem-oriented policing interventions. As a partner in the problem-oriented policing process, 
academics can make a case for using a more rigorous research design to assess the implemented. 
interventions. Many current police executives understand the importance of determining 
“what works” in police crime prevention efforts.” Savvy police executives also understand that 
simple pre—post analyses of time-series data are not scientifically rigorous enough to provide 
strong evidence of a program effect. Indeed, the strong skepticism and dismissive assertions 
made by certain academics in their assessment of the role of innovative policing strategies in the 
1990s crime drop (e.g., Levitt, 2004) made an impression on the police profession. Although 
some police executives (most notably former New York Police Commissioner and former Los 
Angeles Police Chief William Bratton) attacked the credibility of “ivory tower” academics in 
assessing police crime prevention strategies, others such as Redlands, California, Police Chief 


James Bueerman and former Jersey City, New Jersey, Police Chief Francis Gajewski opened 


2. For instance, the Police Executive Research Forum (PERF), which has police chiefs completely comprise 
its board of directors, states “substantial and purposeful academic study is a prerequisite for acquiring, 
understanding and adding to the body of knowledge of professional police management’ as one of its 
organizational principles. PERF also published an edited volume of essays called Police Program Evaluation 
that makes the case that meaningful program evaluation often requires going beyond piecemeal obser- 
vations or simple “before and after” comparisons (Hoover, 1997). 
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their departments to academics looking to conduct rigorous tests of problem-oriented policing 
strategies (e.g., Braga et al., 1999; Weisburd, Morris, and Ready, 2008).? 

Although other important factors undoubtedly exist, ongoing collaborative relationships 
between police executives and academics create opportunities to conduct rigorous evaluations of 
problem-oriented policing initiatives. In an inspiring description of her rewarding experiences 
as an “embedded” criminologist in California’s prison reform efforts, Joan Petersilia (2008) 
rightly observed that “timing is everything’ when working with policy makers and criminal 
justice practitioners. Her observation is particularly salient when considering the legal, politi- 
cal, institutional, and resource constraints inherent in the successful execution of randomized 
controlled trials in criminal justice agencies. 

Throughout the course of a long-standing collaborative relationship with a police executive, 
“timing” directly affected my ability to conduct more rigorous evaluations of problem-oriented 
policing initiatives. In 1997, I formed a collaborative relationship with Edward F. Davis III 
when he was Superintendent of the Lowell Massachusetts Police Department. During the next 
6 years, my colleagues and I worked with the Lowell Police on a series of problem analyses of 
gang violence problems and conducted a quasi-experimental evaluation of a problem-oriented 
intervention to guide their gang violence reduction efforts (Braga, McDevitt, and Pierce, 
2006; Braga, Pierce, McDevitt, Bond, and Cronin, 2008). In 2004, Davis expressed a desire 
to make a substantive contribution to the policing field by conducting a more rigorous test of 
the effects of problem-oriented policing strategies on crime and disorder hot spots (see Braga 
and Bond, 2008). We collaborated on the design of a randomized controlled experiment and 
jointly made a successful argument to the Massachusetts Executive Office of Public Safety for 
research funds. It is important to note that, at that point in time, Davis had served as the chief 
of the Lowell Police for nearly 10 years, crime was low in the city, research funding was avail- 
able, and Davis had gained considerable trust and legitimacy among community members and 
politicians through his commitment to community policing. In other words, the timing was 
ripe for experimentation. 

In December 2006, Davis was appointed Commissioner of the Boston Police Department. 
Boston was facing a troubling increase in gun violence when he took charge of the police force. 
Davis immediately led the department in developing a problem-oriented policing initiative that 
targeted gun violence hot spots in Boston that borrowed heavily from the lessons of the Lowell 
experiment and other evidence-based crime prevention practices. However, at that point in time, 
I could not pursue a randomized controlled trial of the new program. The resulting evaluation 
design was the modal single group before—after design supplemented by qualitative evidence and 
longer time-series data. Indeed, the conditions were different in Boston compared with Lowell 


a few years earlier. His mandate from Mayor Thomas M. Menino was clear. Davis needed to 


3 In his memoir on crime fighting in Boston and New York City, Bratton (1998: 289) stated that “Criminolo- 
gists apparently still had a hard time accepting the reality of our success. | made a conscious decision to 
take on the academics, to challenge conventional wisdom about crime in America and prove that effec- 
tive policing can make a substantial impact on social change” 
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make Boston’s neighborhoods safe by addressing the gun violence wherever it presented itself 
in the city. He was also a new chief who needed to develop the necessary political support to 
permit potentially controversial decisions such as creating treatment and control conditions in 
gun violence hot spots. 

These experiences suggest that academics need to be “commonsense opportunists” who are 
sensitive to the varying needs of their practitioner partners throughout the course of a long-term 
working relationship. Sometimes academics can pursue more rigorous evaluation designs. Other 
times academics will need to respect the political environment and simply provide program 


development and implementation advice that is rooted in solid research evidence. 


Concluding Thoughts 

The purpose of this policy essay was not to make an argument that all evaluations of problem- 
oriented policing need to be randomized controlled trials. Indeed, weaker evaluations can, and 
do, produce important findings. Case studies on the problem-oriented policing process can 
provide rich insights and much-needed guidance to practitioners struggling with substantive 
public safety problems. Nevertheless, the Weisburd et al. (2010) review suggests that the need is 
for more rigorous research in this important area. Weaker evaluations outnumber the stronger 
evaluations by almost five times. Problem-oriented policing scholars, who are mostly likely 
to be in the position to conduct such studies, should recognize the importance of developing 
a stronger evidence base for the approach and foster an environment that encourages more 
rigorous research designs. 

To some observers, long-term working relationships as described in this essay might violate 
the purported scientific necessity to separate program developer and evaluator roles (for discus- 
sion, see Petrosino and Soyden, 2005). However, unless some convincing evidence of widespread 
evaluator bias is associated with such arrangements, these collaborative arrangements seem 
necessary to put academics in the position of being able to conduct higher quality evaluations 
of problem-oriented policing initiatives. As David Olds (2009) suggested in his recent essay in 
support of “disciplined passion,” balancing scientific integrity with the practical challenges that 
are associated with program evaluation in real-world settings needs to be addressed through 
higher standards for reporting trials, better peer review, improved investigator training, and 
rigorous collegial support of those who choose this line of work. 

Although this essay focused on evaluation and assessment issues, it is important to recognize 
that a large gap continues to exist between Goldstein’s (1979) vision of problem-oriented policing 
and its implementation by most police departments. Research has documented that it is difficult 
for police officers to implement problem-oriented policing strategies (Clarke, 1998; Cordner 
and Biebel, 2005; Eck and Spelman, 1987). Deficiencies exist in all phases of the problem- 
oriented policing process. Growing evidence suggests that even the imperfect implementation 
of problem-oriented policing (so-called “shallow” problem solving) generates crime prevention 
gains (Braga and Weisburd, 2006). Improving the problem-oriented policing process, however, 


could produce even stronger crime control effects. Police executives interested in improving the 
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capacity of their agency to deal with crime through more robust problem solving should seek the 
involvement of academics the same way that they seek assistance from community partners and 
other city departments. Beyond evaluation, academics can be helpful in improving the quality 
of problem analysis and broadening the search for alternative responses. 

Federal funding initiatives (similar to PSN) that provide support for academic—police 
practitioner partnerships could raise the quality of problem-oriented policing projects and im- 
prove our knowledge base on effective practice. Unfortunately, the number of academics with 
the experience and expertise in working with police departments on problem analysis, response 
development, and evaluation is relatively small. Although additional funding provides an incen- 
tive, the real challenge is to increase these collaborations by educating police departments on 
the benefits of working with academics and to encourage uninvolved academics to learn more 
about and participate in problem-oriented policing projects. Establishing new and more pro- 
ductive relationships will require flexibility and openness on both parts. Academics cannot be 
condescending in their relationships with the police; they must demonstrate a genuine interest 
in helping the police to be more effective, and they should select research methodologies that 
produce credible results but are not disproportionately demanding given the limits of available 
data as well as the nature of the strategy being evaluated (Goldstein, 2003). The police must 
recognize that quality research takes time and patience, researchers must be objective and require 
facts to support conclusions, and evaluations might produce results that could be interpreted 
as critical of past and current operations (Goldstein, 2003). 

In the brief space given to this policy essay, all factors necessary to produce effective work- 
ing partnerships between academics and police practitioners cannot be covered adequately. It 
does seem obvious that both camps as well as the general public would benefit greatly from a 
sustained collaborative relationship focused on understanding the nature of crime problems 
and on determining what works in dealing with the underlying conditions and situations that 


cause them to persist. 
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